Grid Dynamization

Currently the emphasis is laid on the update rate of indices working with huge data sets, e.g. with all the Internet pages. This article addresses the questions of inverted indices generation, of static data structuers dynamization and finally of dynamic index distribution on more nodes. In each step of upgrading such data structures we describe what changes have to be done and describe benefits (eventually disadvantages) of recently developed data structures. The goal of this paper is to show experimentally that a dynamic inverted index updating distributed to more nodes is quicker then a static one and we can save up to tens percent of read and write operations. Introduction Search engines are systems trying to get answers of questions like “Which documents contain words...” from stored data. We focus on search engines of Internet pages that are more and more significant, so it is important to optimize their search quickness and amount of spent resources. When entering the query, the search engine returns documents sorted by specific criterion (generally a rating function is used, e.g. Google’s PageRank [3; 9]). To optimalize search rate of relevant documents when evaluating the query, search engines use index [4]. Without an index the search engine would have to go trough all the documents what would take a lot of time and computational power in vail. According to [12] there are more then 2.5 billion available pages on Internet. It is unthought to have a search engine without index, going trough all documents would take hours then. Price we have to pay for searching speed is the necessity of periodical index updating and this is the operation which can disallow effective work with an index for a long time. Step-by-step we will build an indexing algorithm of search engine and optimize it. In chapter Inverted index we will describe the procedure of creating an inverted index. In the following chapter Dynamization we attend to dynamize a static data structure and describe dynamic structure characteristics. The chapter Index maintenance details used data structures and their properties like they are applied in Egothor (see [7]). In chapter Index distribution we describe another opportunity how to improve an index actualization properties, and that is by data structures distribution on different nodes. In chapter Simulation we give an account of the simulation program and researched quantities. The chapter Experiments describes the outputs of the simulation program. The last chapter summaries the conclusion resulting from the simulation outputs. Inverted index The inverted index construction can be divided into two steps [6]: First we build a forward index containing all significant words of documents, then we convert it to an inverted index. Consequently the inverted index maps all these words to positions in every document of input set of data files. This data structure provides fulltext search and therefore it is used in data processing systems like search engines. In practice, inverted index is built directly (without the forward index). According to the vector model we define the term weight indicating its significance. All the WDS'07 Proceedings of Contributed Papers, Part I, 116–124, 2007. ISBN 978-80-7378-023-4 © MATFYZPRESS